Data Analysis

Comprehensive Data Cleaning & Exploratory Data Analysis of Job Market Trends

Authors

Affiliation

Connor Coulter

Boston University

Wei Wang

Boston University

Balqis Bevi Abdul Hannan Kanaga

Boston University

1 Import Data

Loaded dataset: (72498, 131)

	ID	LAST_UPDATED_DATE	LAST_UPDATED_TIMESTAMP	DUPLICATES	POSTED	EXPIRED	DURATION	SOURCE_TYPES	SOURCES	URL	...	NAICS_2022_2	NAICS_2022_2_NAME	NAICS_2022_3	NAICS_2022_3_NAME	NAICS_2022_4	NAICS_2022_4_NAME	NAICS_2022_5	NAICS_2022_5_NAME	NAICS_2022_6	NAICS_2022_6_NAME
0	1f57d95acf4dc67ed2819eb12f049f6a5c11782c	9/6/2024	2024-09-06 20:32:57.352 Z	0.0	6/2/2024	6/8/2024	6.0	[\n "Company"\n]	[\n "brassring.com"\n]	[\n "https://sjobs.brassring.com/TGnewUI/Sear...	...	44.0	Retail Trade	441.0	Motor Vehicle and Parts Dealers	4413.0	Automotive Parts, Accessories, and Tire Retailers	44133.0	Automotive Parts and Accessories Retailers	441330.0	Automotive Parts and Accessories Retailers
1	0cb072af26757b6c4ea9464472a50a443af681ac	8/2/2024	2024-08-02 17:08:58.838 Z	0.0	6/2/2024	8/1/2024	NaN	[\n "Job Board"\n]	[\n "maine.gov"\n]	[\n "https://joblink.maine.gov/jobs/1085740"\n]	...	56.0	Administrative and Support and Waste Managemen...	561.0	Administrative and Support Services	5613.0	Employment Services	56132.0	Temporary Help Services	561320.0	Temporary Help Services
2	85318b12b3331fa490d32ad014379df01855c557	9/6/2024	2024-09-06 20:32:57.352 Z	1.0	6/2/2024	7/7/2024	35.0	[\n "Job Board"\n]	[\n "dejobs.org"\n]	[\n "https://dejobs.org/dallas-tx/data-analys...	...	52.0	Finance and Insurance	524.0	Insurance Carriers and Related Activities	5242.0	Agencies, Brokerages, and Other Insurance Rela...	52429.0	Other Insurance Related Activities	524291.0	Claims Adjusting
3	1b5c3941e54a1889ef4f8ae55b401a550708a310	9/6/2024	2024-09-06 20:32:57.352 Z	1.0	6/2/2024	7/20/2024	48.0	[\n "Job Board"\n]	[\n "disabledperson.com",\n "dejobs.org"\n]	[\n "https://www.disabledperson.com/jobs/5948...	...	52.0	Finance and Insurance	522.0	Credit Intermediation and Related Activities	5221.0	Depository Credit Intermediation	52211.0	Commercial Banking	522110.0	Commercial Banking
4	cb5ca25f02bdf25c13edfede7931508bfd9e858f	6/19/2024	2024-06-19 07:00:00.000 Z	0.0	6/2/2024	6/17/2024	15.0	[\n "FreeJobBoard"\n]	[\n "craigslist.org"\n]	[\n "https://modesto.craigslist.org/sls/77475...	...	99.0	Unclassified Industry	999.0	Unclassified Industry	9999.0	Unclassified Industry	99999.0	Unclassified Industry	999999.0	Unclassified Industry

5 rows × 131 columns

2 Data Cleaning & Preprocessing

2.1 Create Derived Columns

Non-null counts: {'INDUSTRY_DISPLAY': 72454, 'SALARY_DISPLAY': 30808}

2.2 Drop Unnecessary Columns

Remaining columns (first 30): ['LAST_UPDATED_DATE', 'POSTED', 'EXPIRED', 'DURATION', 'MODELED_EXPIRED', 'MODELED_DURATION', 'COMPANY', 'COMPANY_NAME', 'COMPANY_IS_STAFFING', 'EDUCATION_LEVELS', 'EDUCATION_LEVELS_NAME', 'MIN_EDULEVELS', 'MIN_EDULEVELS_NAME', 'MAX_EDULEVELS', 'MAX_EDULEVELS_NAME', 'EMPLOYMENT_TYPE', 'EMPLOYMENT_TYPE_NAME', 'MIN_YEARS_EXPERIENCE', 'MAX_YEARS_EXPERIENCE', 'IS_INTERNSHIP', 'SALARY', 'REMOTE_TYPE', 'REMOTE_TYPE_NAME', 'ORIGINAL_PAY_PERIOD', 'SALARY_TO', 'SALARY_FROM', 'LOCATION', 'CITY', 'CITY_NAME', 'COUNTY']

2.3 Handle Missing Values

2.4 Remove Duplicates

Removed 3300 duplicates using ['TITLE', 'COMPANY_NAME', 'LOCATION', 'POSTED']

3 Exploratory Data Analysis (EDA)

3.1 Job Postings by Industry

3.2 Salary Distribution by Industry

3.3 Remote vs. On-Site Jobs

3.4 EDA: Rationale & Insights

3.4.1 Why these visualizations

• Job Postings by Industry: highlights sectors with the most demand.
  
• Salary Distribution by Industry: compares pay levels and spreads across industries.
  
• Remote vs. On-Site: shows flexibility trends in job postings.

3.4.2 Key insights

• A few industries dominate postings, guiding where job seekers should focus.
  
• Salary distributions vary widely by sector, indicating negotiation opportunities.
  
• Remote vs. on-site mix informs geographic/job flexibility strategy.